Network performance and reliability evaluation taking into account attributes other than only the capacities of edges

ABSTRACT

Network performability characteristics with improved accuracy are derived by taking into account, in the various analyzed network failure states, attributes of elements at the logical level other than just the capacities of edges, as well as by taking into account one or more “abstract components,” such as scheduled maintenance, and by using multiple traffic matrices.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional application61/011,814 filed Jan. 22, 2008.

BACKGROUND

The present disclosed method relates to techniques for evaluating theperformance and/or reliability of communications networks, or of anysystem that can be modeled as a communications network.

As communications networks have grown in size and complexity, theevaluation of their performance and reliability has become morecritical. Network service providers usually guarantee a certain level ofservice, in terms of down time, restoration delay, packet delay, etc.,to their enterprise customers. Violating these agreements results in apenalty to the service providers. In turn, the enterprise customersincur business losses if their networks do not perform as expected.There are even some legal requirements on U.S. businesses to be aware oftheir network's “risk profile” (e.g., the US Sarbanes-Oxley act of2002).

Some aspects of the service level have to do purely with reliability,such as downtime, while others are pure performance measures, such aspacket delay. Each can be evaluated or predicted in isolation by anappropriate model and technique, but as networks are becoming larger andmore complex, it is becoming less realistic to consider only purereliability measures or to evaluate performance measures as if thenetwork were always in its perfect (no failure) state. Even though thecomplexity of real networks often requires that performance andreliability evaluations be treated separately from each other, acombined, simultaneous evaluation, known as performability analysis, isvery informative.

Performability analysis consists in characterizing failuresprobabilistically and evaluating a performance measure (or measures)over a large set of network states. There is one “perfect” network statein which all of the network components are assumed to be operatingproperly. Every other network state is characterized by the failure ofone or more of the network components. Among the various network statesthat would be evaluated are a) states in which a particular component isassumed to have failed (in combination with the failure of zero or moreother components) and b) other states in which that particular componentis assumed to have not failed. The probability of various kinds ofcomponents failing is known, based on experience, information frommanufacturers, and statistical sampling techniques, and that probabilityis a factor in the probability of the occurrence of those states inwhich the component in question is assumed to have failed andultimately, then, in computing the overall performability characteristicin question.

An individual network component may actually have more than one failuremode. For example, a link that normally has bandwidth X may fail in sucha way that the available bandwidth is X/2, X/3, etc. Another failuremode is one in which the link has failed completely and thus nobandwidth is available. Any one or these possibilities can occur in somenetwork states. A typical performance measure is packet delay, asmentioned above. Another is the percentage of offered traffic thatcannot be delivered due to the particular failures represented by thenetwork state, referred to herein as “percent traffic lost.” Suchperformance measures are computed based on an assumed traffic matrix,i.e., an assumed amount of traffic demand between eachorigin/destination node pair in the network and, in a typicalapplication, are computed after a network restoration algorithm has beenapplied to the network in the assumed state in order to reroute as muchof the traffic as possible.

The performance measure value computed for each network state is thenmultiplied by the probability of occurrence of that state (which is, inturn, a function, of the probability of failure of the componentsassumed to have failed for that state) and the resultant products areused to derive a performability characteristic. A simple performabilitycharacteristic is what is called the “expectation” of a particularperformance measure, given by the sum of the products just described. Amore sophisticated performability characteristic that can be derivedfrom this analysis is a performability guarantee of the form: “with X%probability, at most Y% of the network traffic will have no path.”

A whole set of performability characteristics can be developed in thisway by considering multiple performance measures for each network state.

For each network state, there is a characterization of the network atthe component level. In order to carry out the analysis just described,a logical level characterization is generated for each network state.The logical level characterization characterizes the network in terms ofnodes and edges that interconnect them. The nodes at the logical levelcorrespond to packet routers or other similar components, whereas theedges represent routes for traffic through the network which maytraverse one or more physical links. A failure of one or more componentsat the component level—including optical fiber spans, routers, repeatersor other physical components—may affect the capacity of one or moreedges at the logical level, capacity being a measure of the amount oftraffic that can traverse an edge as reflected by, for example, thebandwidth of the edge or the bit rate or packet rate that it cansupport. Given an assumed traffic matrix as mentioned above, one canthen use known network analysis techniques to compute the desiredperformance measure—such as, again, packet delay or percent trafficlost—after any restoration algorithm used in the network has been run onto the network state under investigation.

One can then compute desired performability characteristic(s) such asthose mentioned above, i.e., the performance measure's expectation orsome performance guarantee.

SUMMARY

In determining the effects on traffic, at the logical level, of thefailures that characterize a particular network state, prior artapproaches focus on the capacities of edges, as described above.

In accordance with one aspect of our method, we have recognized thatmore accurate performability characteristics can be derived by takinginto account, in the various analyzed network states, attributes ofelements at the logical level other than just the capacities of edges.In particular, we have recognized that more accurate performabilitycharacteristics can be derived by taking into account in the variousanalyzed network states a) attributes of edges for a given network stateother than just their capacities, and b) attributes of logical levelelements other then edges. Examples of a) include an edge's latency andan edge's so-called network routing “cost” or “weight.” An example of b)is whether a particular node is capable of routing additional traffic,such as traffic that might be desired to be rerouted through that nodedue to a failure somewhere else in the network.

BRIEF DESCRIPTION OF THE DRAWING

In the drawing,

FIG. 1 a is a view of the component level of a network for which aperformability characteristic can be derived;

FIG. 1 b is a logical level view of the network of FIG. 1 a;

FIGS. 2 a and 3 a show the network of FIG. 1 in particular respectivefailure states;

FIGS. 2 b and 3 b are logical level views of the network in the failurestates depicted in FIGS. 2 a and 3 a, respectively;

FIGS. 4 a, 4 b, 5 a and 5 b are tables helpful in explaining thecomputation of a performability characteristic;

FIGS. 6 a, 7 a, 8 a and 9 a show the network of FIG. 1 a in particularrespective failure states and illustrate the computation ofperformability characteristics for the network of FIG. 1 a;

FIGS. 6 b, 7 b, 8 b and 9 b are logical level views of the network inthe failure states depicted in FIGS. 6 a, 7 a, 8 a and 9 a,respectively;

FIG. 10 shows three traffic matrices that can be used in computing aperformability characteristic;

FIG. 11 is a flowchart of a typical prior art process for computing aperformability measure; and

FIG. 12 is a flowchart of a process for computing a performabilitymeasure; and

FIGS. 13-26 are figures that help to explain the theoreticalunderpinnings of the invention as presented at the end of the DetailedDescription.

DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENT Basic PerformabilityCharacteristic Computation

FIG. 1 a presents a so-called “component view” of a simple packetcommunications network for which a performability characteristic can bederived pursuant to the principles of our method. It will be appreciatedthat a real-life network would typically be much more expansive, andthus comprising many more components, than shown in this simple example.This component view is also referred to herein as the “physicalnetwork.”

The physical network comprises four packet routers 10 through 13 whichare interconnected by links 14 through 17 as shown. Each of the linksillustratively comprises one or more optical fiber cables.Illustratively, links 14 through 16 each include one optical fiber cable14 a through 16 a, respectively, while link 17 includes two opticalfiber cables 17 a and 17 b. A link may include various other components(not shown) such as one or more repeaters, amplifiers, etc.

Each of the depicted endpoints 18 is representative of a number oforigin and destination endpoints for packet traffic, which are connectedto the routers via respective access networks denoted AC.

Each of the various network components shown in FIG. 1 a and/or theirsub-components is subject to failure with some probability which, as iswell known, can be derived from statistical studies of mean time betweenfailures(MTBF) and mean time to repair (MTTR) of the components. Areal-world network would not only have more components than are shown inthe simple example of FIG. 1A, but also a wider variety of components,each possibly subject to failure with some probability. In addition, itis possible to assess the failure modes of the network more finely byconsidering the failure rates of sub-components, such the line cards andswitching fabrics of the routers 10 through 14.

FIG. 1 b is a logical level characterization of the same network, alsoreferred to herein as the “logical network”. The logical levelcharacterization describes the network in terms of nodes and edges thatinterconnect them. Each edge represents a particular route, or path,that has been established for traffic through the network and maycorrespond to multiple links of the physical network. The logicalnetwork of FIG. 1 b comprises nodes 10 _(L) through 13 _(L)corresponding to routers 10 through 13 of the physical network. Thelogical network also includes particular routes for traffic through thenetwork. It may be seen, in particular, that the logical networkincludes edge 14 _(L) between nodes 10 _(L) and 11 _(L); edge 15 _(L)between nodes 11 _(L) and 12 _(L); edge 16 _(L) between nodes 12 _(L)and 13 _(L); and edge 17 _(L) between nodes 13 _(L) and 10 _(L). Thefour edges 14 _(L) through 17 _(L) illustratively correspond to physicallinks 14 through 17, respectively. The logical network also includesedge 19 _(L) for traffic between nodes 11 _(L) and 13 _(L). As seen fromFIG. 1 a, there is no physical link directly connecting routers 11 and13. Rather the traffic associated with edge 19 _(L) actually traverseslinks 15 and 16 of the physical network. FIGS. 1 a and 1 b represent afirst state of the network-the so-called “perfect” network state inwhich there are no failures.

A failure in a physical component of the network at the component levelhas implications for the routing of traffic over the edges of thelogical network. For example, FIG. 2 a shows a network state in whichlink 16 has totally failed, thereby reducing the bandwidth capacity ofboth edges 16 _(L) and 19 _(L) to zero, as shown in FIG. 2 b. Arestoration algorithm may reroute some or all of the traffic betweennodes 11L and 13L on links 14 and 17. Thus, after restoration, edge 19Lmay be regarded as restored, possibly to its full capacity. Similarly,some or all the traffic between nodes 12L and 13L that went over link 16may be rerouted over links 15, 14, 17, and thus edge 16L may beconsidered restored, possibly to its full capacity.

The physical network of FIG. 1 a has many more possible networkstates—each corresponding to a respective logical network—depending onwhich components and/or combinations of components might fail at anygiven time. For example, FIG. 3 a shows another network state, in whichcable 17 b—one of the two cables of link 17—has failed, resulting inedge 17 _(L) having only 50% of its normal capacity. Unlike the case ofFIGS. 2 a/2 b, edge 19 _(L) is unaffected by this particular failuresince it does not traverse the physical link 17.

Multiple concurrent failures of components of the network are alsopossible, and the resulting logical networks would represent networkstates in which the paths take all such failures into account. Examplesof that are given below.

FIG. 11 is a flowchart of a prior art process for computingperformability characteristic(s) of a network such as the network ofFIG. 1 a.

The process begins (as indicated at 1101) by identifying all thecomponents of the physical network that are subject to failure. Alsoidentified are their respective failure modes and the effects that eachpossible failure mode will have at the logical level. For example, wesaw above that among the effects of the failure of link 16 is that thebandwidth capacities of both edges 16 _(L) and 19 _(L) are reduced tozero.)

A logical level “master” graph is then generated (1104), showing all thenodes and edges of the network at the logical level. The graph of FIG. 1b is such a master graph.

A traffic matrix must also be assumed (1105). That is, we must assumesome level of traffic between the various origin-destination node pairs.This statistical model of the offered traffic is typically derived fromactual network measurements. One could also use a hypothetical trafficmatrix in order to compute desired network performabilitycharacteristic(s) under an assumed traffic demand.

The process of identifying the network states then begins (1106). Asalready indicated, each network state is characterized by a set of oneor more failures of one or more physical components. The process willtypically not actually consider all of the network states in carryingout the performability characteristic computation, because there aretypically too many states to consider in order for the computation to becarried out in a reasonably short time. This is discussed further below.

The identified state is now considered (1109). We first generate alogical level graph for that state (1113) by applying to the mastergraph the effects at the logical level of the failures assumed for thisstate. The graph of FIG. 2 b is an example of a logical level graph forthe network state that assumes the failure of link 16.

The real-life network being modeled may employ a restoration algorithmthat attempts to reroute traffic after network failures. In that case,the particular restoration algorithm that is used in that network isapplied (1120) to the logical level graph of the network for the assumednetwork state before computing the desired performance measure(s), suchas packet delay or percent traffic lost. The reason is that theperformability characteristic computed for the network should reflectthe extent to which the network is able to recover from failures and totherefore ameliorate the effects of those failures.

We can now compute the performance measure(s) of interest, e.g., packetdelay or percent traffic lost using known network analysis techniques(1124).

The performance measure thus computed is only for a particular networkstate. What we care about is an overall performability characteristic ofthe network, taking into account a) the various ways in which componentscan fail, both singly and in various combinations, and b) the failureprobabilities. Accordingly, if there are more states to be considered(1131), we consider the next state (1106) and carry out the performancemeasure computation for that state, and so on, after which the desiredperformability characteristic(s), such as the expectation of a givenperformance measure, can be computed (1135).

It is possible to compute the desired performance measure, and thus theoverall performability characteristic, over the entire set of possiblenetwork states. However, as noted above, there are typically too manystates to consider in order for the computation to be carried out in areasonably short time. What one can do is to consider the network statesat 1109 and 1132 in order of decreasing probability and update theperformability characteristic(s) of interest after the performancemeasure has been calculated after every N states, where N is anyconvenient integer. The successive values of each performabilitycharacteristic will begin to converge after some number of networkstates has been considered and one can use, for example, the rate ofconvergence to determine when the last-computed performabilitycharacteristic value is “close enough” for the purposes at hand so thatone need not take further states into account.

FIGS. 4 a and 4 b illustrate the computation of a performabilitycharacteristic F for a simple network assumed to have only threecomponents that may fail, denoted C1, C2 and C3. As shown in FIG. 4 a,components C1, C2 and C3 have probabilities of failing P1, P2 and P3,respectively. Assuming that each of the components C1, C2 and C3 canfail independently, the overall network has seven failure states—denoted1 through 7 in FIG. 4 b—in which any one, any two, or all three of thecomponents C1, C2 and C3 have failed, the bit value “1” or “0” in aparticular column respectively indicating that the associated componenteither has or has not failed. There is also the perfect state, numbered0. The probability of the network being in any state is found in theform of a product of probabilities determined by the bit vectorcorresponding to the state: in forming the product, if a bit is 1 wetake the failure probability of that component, and if it is 0 we take 1minus the failure probability. For example, state 5 in which componentsC1 and C3 have failed, is defined by the bit vector 101, so itsprobability is SP5=P1×(1−P2)×P3.

Associated with network states 0 through 7 are respective performancemeasure values F0 through F7 of a performance measure F, F being, forexample, “percent lost traffic.” The values of F0 through F7 arecomputed in a manner well known to those in the art, based on a) thelogical network associated with any given failure or combination thereofand b) an assumed traffic matrix. As already noted, the restorationalgorithm used by typical real-world networks to re-route traffic inresponse to traffic-disruptive component failures would typically beapplied to the logical network associated with each network state beforecomputing the performance measure values F1 through F7.

The assumed performability characteristic of interest is the expectation{tilde over (F)} of the performance measure F. As shown in FIG. 5 b, theexpectation {tilde over (F)} is given by the sum of the products of a)the probability of each given state occurring with b) the performancemeasure value computed for that state. Performability characteristic Fprovides an indication of the extent to which a particular aspect ofnetwork service will be affected by the failure of network components.If F is a measure of something undesirable, the smaller the value ofperformability characteristic {tilde over (F)} the better. If thenetwork operator regards the value of performability characteristic{tilde over (F)} to be unacceptably high, the network operator may takesuch steps as reconfiguring its network or providing the network withadditional redundancy. As discussed in further detail below, an evenmore useful performance characteristic than just the expectation ofperformance measure F is a performability guarantee that, in thesimplest case, could take the form “with 99.9% probability, at most 5%of the total traffic will not be able to be routed to its destination.”

FIGS. 5 a and 5 b illustrate another computation of a performabilitycharacteristic F for the same network assumed for the illustration ofFIGS. 4 a and 4 b except now taking into account the possibility ofcomponents with multiple failure modes. Specifically, it is assumed forthis example that while components C1 and C2 each still have only onefailure mode, C1 ₁ and C2 ₁, respectively, component C3 has threefailure modes C3 ₁, C3 ₂ and C3 ₃, which are mutually exclusive. Failuremodes C1 ₁ and C2 ₁ have respective probabilities P1 and P2 and failuremodes, C3 ₁, C3 ₂ and C3 ₃ have respective probabilities P3 ₁, P3 ₂ andP3 ₃. Thus, in addition to the perfect state, there are, overall, 15network failure states, as shown in FIG. 5 b, having the stateprobabilities SP1 through SP15, performance measure values F1 throughF15 and performability characteristic {tilde over (F)} as shown.

In determining the effects on traffic, at the logical level, of thefailures that characterize a particular network state, prior artapproaches focus on the capacities of edges, as described above.

In accordance with one aspect of our method, we have recognized thatmore accurate performability characteristics can be derived by takinginto account attributes of elements at the logical level other than justthe capacities of edges. In particular, we have recognized that moreaccurate performability characteristics can be derived by taking intoaccount in each analyzed network state a) attributes of edges for agiven network state other than just their capacities and b) attributesof logical level elements other then edges. These will now be discussedin turn.

Attributes of Edges Other Than Just Capacities

FIGS. 6 a and 6 b illustrate taking into account attributes of edgesother than just their capacities in deriving the various networkstates—specifically, in this example, latencies of edges.

Latency is the time it takes for a packet to travel from its point oforigin to its point of destination. In the example of FIGS. 6 a and 6 b,optical cables 17 a and 17 b have latencies X and Y, respectively.Illustratively, traffic is equally likely, on average, to be on cable 17a or 17 b, and so the latency of the overall link 17 is the average[X+Y]/2. Assume that a repeater associated with optical cable 17 b hasfailed, taking cable 17 b out of service. The latency of the link 17thus changes from [X+Y]/2 to simply X, i.e., the latency of cable 17 a.In accordance with our disclosed method, various network states thatwould be considered include states in which a) the latency of link 17 isits “normal” latency associated with all of its constituent cables beingoperative (in this example [X+Y]/2) and, alternatively, b) the latencyof link 17 is its latency under the assumption of the failure of one ormore of its constituent optical cables (in this example, X). Sincelatency is a factor in how well a network is performing, then thecomputation of various performability characteristics can be moreaccurately computed, pursuant to this aspect of our disclosed method, byincluding such network states when computing various performancemeasures that are involved in performability characteristics such as theaverage time for the network to route packets end-to-end.

The length of a route can also affect various measures of networkperformance. Thus the network states that are evaluated in computingvarious performability characteristics pursuant to this aspect of ourdisclosed method may include states in which the length of the edges ofthe logical network differ from “normal” as the result of failures.

Another example related to latency arises in the context of routesbetween nodes in an underlying network layer that is not explicitlyincluded in the model, such as a SONET ring. Normally, traffic betweentwo points A and B is routed in a particular direction around the ring.If a component in that direction should fail, the traffic between A andB could be re-routed in the other direction, which could be the long wayaround the ring, thus increasing the latency of the edge between nodes Aand B of the ring.

FIGS. 7 a and 7 b further illustrate the taking into account ofattributes of edges other than just their capacities in deriving thevarious network states—specifically, in this example, the so-called“cost” (also called the “weight”) of edges of the logical network.

Links in a network are often assigned a so-called administrative costthat is a function of, for example, the physical length of the link.Algorithms that determine routes through the network can take link costsinto account in doing so, with the goal being to develop routes thatminimize the use of network resources or to maximize throughput or toachieve some other goal. The routing algorithms are designed such thatthey tend to avoid routing traffic over links having relatively highcosts. In addition to assigning baseline, or fixed, administrative coststo links, network management software can artificially increase a link'scost if for any reason it is desired to “discourage” a routing algorithmfrom routing additional traffic over that link.

For example, if one or more of a router's line cards fail, or ifmaintenance is being performed on the router, its packet-handlingcapabilities are reduced. In such a situation the network operator maysimply take the router out of service for the period of repair ormaintenance. A better operations policy is to discourage the routing ofnetwork traffic via the impaired router by increasing the administrativecost associated with all links to which that router is connected; thismakes the routing algorithms steer traffic away from the router inquestion. An example is shown in FIGS. 7 a and 7 b. It is assumed thatone of the line cards of router 11 has failed, with the result thatnetwork management software increased the costs associated with links 14and 15 by an amount Δ.

Increasing the link costs is a better network operations policy becauseit effectively removes the router in question from the network, butstill allows it to be used in the event that a failure elsewhere in thenetwork is such that a demand affected by it has no path to itsdestination other than (being rerouted) through this impaired router. Asan example, consider router 11L in FIG. 7 b and assume that while thisrouter is impaired, link 16 in FIG. 7 a fails, thus removing edge 16Lfrom the network of FIG. 7 b. In this situation the only path for thedemand from router 13L to 12L is via router 11L, despite the increasedcost of this path. A performability evaluation method which, unlike thisaspect of our disclosed method, does not allow increases of edge costsas the result of component failures, would represent this situationsimply by removing router 11L from the network of FIG. 7 b, and woulderroneously conclude that the demand from 13L to 12L would be lost.

The above example is one in which an attribute of a link other than justits capacity is affected not by a failure on the link itself, but ofsome other component of the network. For another example, consider theloss of packets in a network employing radio, microwave, or laser links.Each link has a “packet loss probability” attribute, whose value isaffected by components representing atmospheric conditions or conditionsin space. The overall loss in the network is a function of these linkloss attributes. The loss probability attribute may also depend on theavailable buffer (queue) space of routers to which the link isconnected. Thus, if one of N buffers in a router fails, there is anincreased probability of loss in the associated links since theprobability that buffers will overflow, and thus the rate at whichpackets are dropped, is increased.

FIG. 12 is a flowchart of a process for computing a performabilitymeasure pursuant to the principles of our method. The process isstructured similarly to the prior art process shown in FIG. 11 anddiscussed above. Indeed, each of the steps in FIG. 11 has acorresponding step in FIG. 12 and the last two digits of the referencenumerals in the two FIGS. are the same for corresponding steps.

In accordance with this aspect of our disclosed method, however, step1213 takes into account changed “graph element attributes” rather thansimply changed link capacities at 1113 of the prior art method of FIG.11. FIG. 12 thus reflects that our method allows for the possibility ofconsidering attributes of edges other than just their capacities. Otherdifferences between the processes of FIGS. 11 and 12 are discussedbelow.

Attributes of Nodes

The above examples are ones in which, pursuant to one aspect of ourdisclosed method, one or more attributes of edges other than just theirrespective capacities are taken into account in defining the networkstates. We saw how those edge attributes might be affected by a failurewithin the associated physical link and/or within some other networkcomponent.

In accordance with another aspect of our disclosed method, however, thenetwork states may include states in which a failure affects theattributes of a node. An example of such a state is depicted in FIG. 8 ain which it is assumed that router 10 has failed in such a way that itcannot accept any new connections, even though it can continue to routepackets of connections that were established before the failure. Thus inthe logical network of FIG. 8 b no new circuits can be routed throughnode 10 _(L). Assume that at the same time as this failure, link 16 hasfailed. Assume also that logical path 19 _(L) is routed via the physicalroute that involves links 16 and 15. If the aforementioned attribute ofnode 10 _(L) were not taken into account, a performance measure such aspercent traffic lost would be inaccurate because it would, erroneously,not be negatively impacted by the failure of path 16 _(L). This isbecause the network's restoration algorithm applied to the logicalnetwork would assume a successful rerouting of that traffic from node 13_(L) to node 11 _(L) over edges 17 _(L) and 14 _(L). In accordance withthis aspect of our method, however, the network states include states inwhich node 10 _(L) is unable to accept any new connections. In thisparticular example, then, the performance measure for this network statewill, more accurately, reflect the fact that there is no available pathbetween nodes 13 _(L) and 11 _(L), which will negatively impact on thepercent traffic lost performability characteristic. Nice!

FIG. 12 reflects this aspect of our process, as well. As before, notethat the method, at 1213, takes into account changed “graph elementattributes” rather than simply changed link capacities at 1113 of theprior art method of FIG. 11.

Abstract Components

In accordance with an aspect of our method, we have recognized that moreaccurate performability characteristics can be derived by including inthe component-level characterization of the network one or more of whatwe refer to as “abstract components.” These are aspects of the networkand/or its operation that can have the same kind of effect at thelogical level—and thus on traffic and the performance measure—as anactual physical component failure.

An example, illustrated in FIG. 9 a, is a “scheduled maintenance”subcomponent of the network's routers. It is assumed in this examplethat a router under maintenance either has certain of itstraffic-handling capabilities reduced or is taken out of servicealtogether, this having the same effect at the logical level as a totalor partial failure of the router. The legend “includes scheduledmaintenance subcomponent” is shown in FIG. 9 a only for router 11 butwould typically apply to all of the network's routers.

The scheduled maintenance of routers may be handled by network managersby artificially increasing to a very high number the “cost” assigned toall links connected to the router, so that the only way any traffic willgo through this router is if a failure somewhere else makes paths usingthis router unavoidable. In the example where router 11 is undergoingmaintenance, the costs of links 14 and 15, again assumed to be X and Y,would be increased by some amount Δ when the restoration algorithm isrun on the logical network of FIG. 9 b and the performance measure underconsideration is computed.

As to the “probability” associated with the scheduled maintenancesubcomponent of, say, router 11, assume that scheduled maintenance iscarried out once per week for two minutes. The percentage of time thatthe router is in this effectively “failed” mode is then equal to thefraction of time that the router undergoes scheduled maintenance. Sincethere are 10,080 minutes in a week, that fraction is 2/10,080=0.000198,which is then used as the “probability” of this “failure” occurring.Advantageously, this abstract failure mode of the network's routers canbe regarded as being just another one of the routers' failure modes, andamong network states that are evaluated when computing a performabilitycharacteristic are network states in which a router has “failed” in thisway.

FIG. 12 reflects this aspect of our process, as well. Note that at 1201the process allows for the possibility of abstract components likescheduled maintenance.

Multiple Traffic Matrices

In accordance with another aspect of our method, we have recognized thatmore accurate performability characteristics can be derived by computinga performance measure using not a single traffic matrix but, rather, twoor more traffic matrices.

As an example, FIG. 10 shows three traffic matrices I, II and II for anetwork illustratively having five origin/destination nodes, denoted 1through 5. These traffic matrices could be derived from a networkoperator's observation that traffic patterns fall into three particularcategories for the network in question: a) non-holiday weekdays from 7AM to 6 PM (matrix I); b) non-holiday weekdays from 6 PM to 7 AM (matrixII), and c) weekends and holidays (matrix III). The three differenttraffic matrices depicted in FIG. 10 are associated with these threetime periods which, depending on what counts as a “holiday,” can becomputed as comprising about 32%, 38% and 30%, respectively, of thehours in a year.

The amount of traffic from node 1 to nodes 2, 3, 4 and 5 during the timeframe associated with matrix I is denoted b1, c1, d1 and e1,respectively. It is assumed that traffic between nodes may beasymmetric—that is, the amount of packet traffic from one node toanother may be different in the reverse direction. Thus the amount oftraffic to node 1 from nodes 2, 3, 4 and 5 during the time frameassociated with matrix I is indicated separately, being denoted f1, k1,p1 and u1, respectively.

The traffic amounts indicated in matrices II and III follow a similarlabeling scheme.

A first way of taking multiple traffic matrices into account is tocompute the value of the performance measure F for each network stateusing each traffic matrix in turn. One can then amalgamate the resultingperformance measure values F into a composite performance measure valuefor the network state under consideration, weighting them in accordancewith the length of time over which the traffic associated with thevarious traffic matrices is assumed to subsist.

This first approach is illustrated in FIG. 12. A first traffic matrix isassumed at 1218 and the performance measure is computed for the presentstate based on the logic level graph and assumed traffic matrix (1224).The next traffic matrix, if there is more than one, is then considered(1227) and the performance measure is computed with that traffic matrix,and so forth. After the performance measures have been computed for eachtraffic matrix, they are amalgamated into a composite performancemeasure for the network state in question by weighting them by, in thisexample, the factors 0.32, 0.38 and 0.30 (1228).

Thus if a particular performance measure F3 for a particular state iscomputed to be F3 ¹, F3 ² and F3 ³ for traffic matrices I, II and III,respectively, then the composite value of the performance measure F3 forthe state in question would be given by (0.32F3 ¹+0.38F3 ²+0.30F3 ³).

A second approach for taking multiple traffic matrices into account isto define an abstract component—per the discussion of abstractcomponents above—and to make the traffic matrix part of the networkstate. In this case the abstract component would be a traffic component.In any given network state the traffic component will be in (in thisexample) one of the three modes (corresponding to the three trafficmatrices). The probability associated with each traffic mode used forcomputing the probability of the overall network state is the associatedpercentage of time that network traffic is characterized by thatparticular matrix.

The second approach may yield the same computed performabilitycharacteristic as the first if all of the possible network states areconsidered in the computation. If significantly less than all of thepossible network states are considered in the computation, a different,but probably statistically insignificant, value of the performabilitycharacteristic might be expected.

There are, however, certain advantages to using the second approach. Oneof them is that if the implementing software allows for abstractcomponents, as discussed above, the inclusion of different trafficmatrices in the computation is readily accomplished without having togenerate program code specifically directed to multiple-traffic trafficmatrix considerations.

Moreover, in certain situations the second approach will yield a moreaccurate computation. As an example of that, assume that the software inthe network's routers gets upgraded only when the network traffic is low(say between 2 AM and 4 AM). In that case, we want the abstractcomponent “router software upgrade” to be correlated with low traffic.Indeed, the implementing software will typically already have beenconfigured to allow for correlated component modes. Accordingly, thosestates in which the router is in software upgrade mode will always havethe abstract traffic component in its mode associated with the timeframe that encompasses 2 AM to 4 AM. That is, we will not consider anynetwork state in the overall computation in which the router is insoftware upgrade mode but the traffic matrix is for a time frame thatdoes not encompass 2 AM to 4 AM. This is correct to do because no such astate ever occurs in the postulated scenario. If, by contrast, the firstapproach, as shown in FIG. 12 were used, F would be computed using suchnon-existent states and, as such, would be less accurately determined.

Theoretical Underpinnings 1 Introduction

The balance of this Detailed Description presents some theoreticalunderpinnings of our disclosed method as well as further examples.

2 Network Performability Evaluation

We describe the problem of network performability evaluation in anabstract manner and, for clarity of exposition, restricted to networkswith only binary components. More complex multi-mode models arediscussed hereinbelow

2.1 The Theoretical Problem

Let C={c₁, . . . , c_(n)} be a set of “components”, each of which iseither in a working or in a failed state. A component may, or may not,correspond to an actual network element. In general, it is best thoughtof as representing some failure mechanism. Components associated withnetwork elements may represent hardware or software that can experiencefailure. Other components may represent more complex possiblecombinations of events, such as a natural disaster, that bring down allnetwork equipment in a certain geographical area. The main assumptionabout components is that c_(i) fails with probability q_(i)independently of any other components.

A state of the network is an assignment of a state to each component andcan be thought of as an n-bit vector. The set of all network states S(C) has size 2^(n), and the probability of a particular state is theproduct of the state probabilities of the n components. Let F (•) be avector-valued performance measure (function) defined on S (C), mappingeach state to an m-tuple of real numbers; see §4 for examples.

The most basic characteristic of F is its expected value over the spaceof network states:

$\begin{matrix}{\overset{\_}{F} = {\sum\limits_{s \in \; {S{(C)}}}{{F(s)}{{\Pr (s)}.}}}} & (2.1)\end{matrix}$

Our main purpose is to compute more sophisticated characteristics of Fthan just its expectation. These are performability guarantees which, inthe simplest case have the form, e.g., “with 99.9% probability, at most5% of the total traffic is down,” and in a more complex case the form“with 99.9% probability at most 5% of the total traffic is down, andwith 99.99% probability at most 10% is down.” Formally, these guaranteesare statements of the type

Pr(F<x ₁)>p ₁ , Pr(F<x ₂)>p ₂, . . . , or

Pr(F>y ₁)<q ₁ , Pr(F>y ₂)<q ₂, . . .   (2.2)

that apply to the entire space S. These guarantees, referred to as“multi-part performability specifications” are much stronger and moreuseful than just the expectation (2.1). For instance, they can be usedto set, or to check, service-level agreements (SLAs) for the network. Analternative assessment uses multi-attribute risk, examined in §4.4.

What is the meaning of F if the components in C represent physical unitsthat can fail and be repaired? If a component is represented by a2-state Markov process, then S is the state space of a large Markovchain, defined by the parallel composition of the n independent 2-stateprocesses. Then it is easy to see that the limiting average of F overtime as the network moves through its states is precisely the F of(2.1), if we take the failure probability of c_(i) to be thesteady-state probability that the ith Markov chain is in its failedstate. When the individual Markov processes are ergodic (always true forthe network processes we see in practice), then the probabilities in(2.2) have similar interpretations as limiting time averages, i.e.,yield the long-term fraction of time over which the correspondingstatements, F<x or F>y, are true.

2.2 Approximate Evaluation

Since the size of S (C) is normally very large, the straightforwardcomputation of (2.1) or (2.2) is generally prohibitive. The complexityof the Performability Evaluation problem

Given a set C and a polynomial-time computable function F (•) defined onS (C), compute the expectation F defined by (2.1), depends on F. Evenfor very simple F this problem is computationally difficult. Forexample, it contains as a special case the #P-complete problem ofTwo-terminal Network Reliability evaluation, which asks for theprobability of there being a path between vertices s and t of a graphwhose edges are subject to failure with some probability. Various waysof approximating F are known in the literature. They can be classifiedinto (a) upper and lower bounds, for certain F such as connectivity, andspecial network structures (b) “most probable states” methods, (c) MonteCarlo sampling approaches and (d) probabilistic approximation algorithmsfor simple F. Methods of types (a) and (b) produce algebraic bounds onF, while (c) and (d) yield statistical bounds. Here we adopt the “mostprobable states” methods, which are based on the observation that if thecomponent failure probabilities are small, most of the probability massof S will be concentrated on only a few out of all the possible states(equivalently, there will be many failure states of low probability).Then, the challenge is to conceive methods to allow tractablecomputation of these states within a probabilistic, pre-specified“precision”. Let ω and α be the smallest and largest values of F over S.Given a set C, suppose we generate the k highest-probability states, andthey have total probability P. Then we have algebraic lower and upperbounds on F

$\begin{matrix}{{{\overset{\_}{F}}_{l} = {{\sum\limits_{i = 1}^{k}{{F( s_{i} )}{\Pr ( s_{i} )}}} + {( {1 - P} )\omega_{i}}}},{{\overset{\_}{F}}_{u} = {{\sum\limits_{i = 1}^{k}{{F( s_{i} )}{\Pr ( s_{i} )}}} + {( {1 - P} ){\alpha.}}}}} & (2.3)\end{matrix}$

By viewing states as sets of failed components, F is monotone increasingif F (s)≦F (s′) whenever s⊂s′, and monotone decreasing if F (s)≧F (s′).It is possible to improve the bounds of (2.3) for monotone measures F.The particular performance measures we use here are defined in §4 and weshow that some, but not all, are monotone.

A generalization of most probable states is the notion of “mostimportant states”, i.e. those which contribute the most to the value ofF. We did not adopt this—in principle more powerful—approach, becausedetermining the states on which the function F takes large values (a) isdifficult for the F and types of network of interest here (cf. §4), and(b) depends strongly on the network design and so does not generalize tothe degree we require. Further, for the kinds of networks we willencounter in practice, our experience with the most probable statesmethod is that usually the values of F and the probabilities of thestates are such that states are in fact generated in most importantorder.

State generation algorithms There are two fundamentally different typesof algorithms for generating states in order of probability. One type ofalgorithm views the space S as a partially-ordered set (lattice), whilethe other treats it as a binary tree. Both these algorithms work only onsystems of binary components, and they are best suited to differentproblem domains. For a monotone F, the second algorithm produces betterbounds on F than those of (2.3). Equivalently, this algorithm will needto explore fewer network states to achieve a given accuracy on F.However, the algorithm works only for monotone F, and it consumes alarge amount of memory, which depends strongly on the size of C; thisusually renders the algorithm impractical on large network models. Whenwe have non-monotone measures and large networks, we use the firstalgorithm that is applicable to any measure, monotone or not, at theexpense of the looser bounds (2.3), but whose memory usage is a veryweak function of the number of components in C. Finally, for models thatinclude components with more than two failure modes, we present a newalgorithm described in §2.3.

Bounds on the CDF So far we have discussed bounds only on theexpectation of a measure F. The same techniques allow us to derivebounds on the performability guarantee Pr(F≦x), (CDF of F) for a givenx. Let the notation [C] yield 1 if the condition C is true and 0otherwise. Then

${\Pr ( {F \leq x} )} = {\sum\limits_{s \in \; S}{\lbrack {{F(s)} \leq x} \rbrack {\Pr (s)}}}$

is a sum of the form (2.1), so bounds of the type (2.3) can be found forit as well (recall that P is the probability of the khighest-probability states):

$\begin{matrix}{{\sum\limits_{i = 1}^{k}{\lbrack {{F( s_{i} )} \leq x} \rbrack {\Pr ( s_{i} )}}} \leq {\Pr ( {F \leq x} )} \leq {{\sum\limits_{i = 1}^{k}{\lbrack {{F( s_{i} )} \leq x} \rbrack {\Pr ( s_{i} )}}} + 1 - {P.}}} & (2.4)\end{matrix}$

2.3 A Hybrid State Generation Algorithm

Our experience indicates that many practical telecommunications networkscan be adequately modeled by binary components together with arelatively small number of components with multiple failure modes. Thereis no satisfactory state generation algorithm for such a system. To ourknowledge, the so-called GC algorithm is the most efficient for systemsof purely binary components, but cannot handle any non-binarycomponents. Other algorithms work with multi-mode components, includingbinary, but they are not nearly as efficient as the GC algorithm in thepurely binary case.

A method is known for combining an arbitrary binary algorithm (forbinary components only) and an arbitrary multi-mode algorithm (for anycomponents) into a hybrid algorithm which works for combinations ofbinary and non-binary components, but is particularly efficient formostly binary systems of components. By “mostly binary” we mean suchthat the number of binary failure modes is at least as large as thetotal number of multi-mode failure modes. All state generationalgorithms that we are aware of use a priority queue which storesnetwork states according to probability, and a set of rules forgenerating next states from a current state. The hybrid algorithmfollows the same pattern, and is described in FIG. 13; the concept of1-states is central to the algorithm, and will be defined shortly. Notethat the multi-mode algorithm M chosen for incorporation into the hybridscheme needs a slight generic modification; we denote the modifiedalgorithm by M′.

Efficiency We now present a more complete discussion of the efficiencyof the hybrid algorithm. Given the components C={c₁, . . . , c_(n)}, wedefine a network state to be an n-tuple of (component id, mode) pairs.If a component has μ failure modes, we denote their probabilities by p₀,p₁, . . . , p_(μ), where p₀ is the probability of the working(non-failure) mode. We assume that each component's modes are arrangedin order of decreasing probability, so p₀>p₁≧p₂≧ . . . . A 1-state is astate in which all components are in mode 1, their most probable failuremode.

Notice that for a given set C and ε>0 any algorithm, hybrid or not, forgenerating states in order of probability, will produce exactly the samesequence of states (ignoring the ordering of states with equalprobabilities). So the algorithms M and (B, M′) will generate the sames₀, s₁, s₂, . . . when run on C. The gain of the hybrid (B, M′) over theM comes from lines 6 and 7 in FIG. 13, and is based on the assumptionthat B generates 1-states more efficiently than M. The gain isquantified by the fraction of states among s₀, s₁, . . . that are1-states. This fraction is difficult to calculate, but, intuitively, itcan be claimed to be close to 1 as follows: Let S_(k) be the set ofstates consisting of k failed components. Then

Claim 1 A 1-state in S_(k) involving a particular set of k components ismore probable than any other state in S_(k) involving these components.

Claim 2 For small k, a lot of the states of S_(k) are 1-states. Claim 1follows from the assumed ordering of the mode probabilities. Tosubstantiate Claim 2, suppose C contains b binary components and mmulti-mode components, which have a total of M failure modes. 1-stateswith k components can be formed by choosing k out of b+m failure modes.Any state with k components can be formed by choosing k out of b+Mfailure modes, but choices involving two or more modes among the M thatbelong to the same component will be invalid. Therefore

$\begin{matrix}{\frac{{{no}.\mspace{11mu} {of}}\mspace{14mu} 1\text{-}{states}\mspace{14mu} {in}\mspace{14mu} S_{k}}{{{no}.\mspace{11mu} {of}}\mspace{14mu} {all}\mspace{14mu} {states}\mspace{14mu} {in}\mspace{14mu} S_{k}} > \frac{\begin{pmatrix}{b + m} \\k\end{pmatrix}}{\begin{pmatrix}{b + M} \\k\end{pmatrix}} > {( \frac{b + m - k + 1}{b + M - k + 1} )^{k}.}} & (2.5)\end{matrix}$

In a “mostly binary” network as defined above we have b>M, and forrelatively small k the ratio in (2.5) is comparable to 1, approaching 1as b becomes larger with respect to M.

E.g., for k≦3, if M=b/10, the ratio is ≧0.75 for b≧100.

Now if M is comparable to b, the fraction of 1-states in S_(k) is goingto be smaller, but in any practical situation, we only generate a smallsubset of states. Because the 1-states have high probabilities, we canmake an intuitive argument that 1-states form a large percentage of thegenerated states s₀, s₁, . . .

Suppose we sort all b+M failure modes in the decreasing order ofprobability. A network state corresponds to a subset of these modes andits probability is proportional to the product of the corresponding modeprobabilities. Let us consider the states in S₃ in a system with b+m=500components. There are more than 2.5·10⁶ triplets formed out of the first250 modes, and in any practical situation we expect most of the statesto be formed out of these first 250 modes. This can be seen as follows:the mode probabilities of any c_(i) are in decreasing order, so a largenumber of the first 250 modes will be mode 1. If, for example, 80% ofthem are mode 1, then the fraction of 1-states among all triplets formedout of the first 250 modes is about (0.8)³>½ (think of selecting a modeat random out of 250).

In other words, more than half of the states of S₃ that are generatedwill be 1-states. FIG. 14 substantiates this argument by showing theexperimental maximum queue length as a function of the number ofgenerated states in a network that is not “mostly binary”. The blackpoints show the behavior of the binary algorithm on a system with 500purely binary components. The red points (lower) show the behavior ofthe hybrid algorithm on the same system, but where 50 of the componentshave been made to have 10 modes each. Each algorithm is run with 50randomly-selected values of p₀ in [0.999, 1), and the failure modes ofthe multi-mode components have equal probabilities. It can be seen thatthe efficiency of the hybrid algorithm is competitive with that of thebinary algorithm.

2.4 General Context

To gain some perspective on what has been said so far and to understandsome of the compromises involved, it is useful to considerperformability evaluation in a general setting. Ideally, one would liketo model the network under consideration as a system of (perhapsinteracting) components c₁, . . . , c_(n). Each component would berepresented in some powerful formalism, e.g. as a stochastic Petri netor a stochastic automaton. Such component models would allowrepresenting aspects of the real network that were idealized in thedescription of §2. These formalisms also permit the specification of aperformance measure, often called a “reward”, on the states of thesystem. There are well-developed methods for analyzing such systemsexactly when the number of components is small, and approximately whenit is large. In the best case, the combination (composition) of thecomponent models is translated into a large Markov chain, which is thensolved numerically, either exactly, or with some state-space “pruning”.When reduction to a Markov chain is not possible, the resulting model isa discrete-event system. Although some special cases (regenerative) canbe solved numerically, such a model must, in general, be dealt with viasimulation. In all cases, the difficulty of the analysis increases withthe size of the model and with the precision of the results that onewants: averages of measures are the easiest to compute, probabilitydistributions are more difficult, and probability densities are evenmore difficult. Also, it is much easier to do all this with the systemin steady state, as opposed to considering its evolution (dynamics) intime.

There are two difficulties in applying such methods to our networkperformability evaluation problem. First, the number of components islarge: n≈700 is not unusual, leading to a large state space. Second, ourperformance measure is complex: it involves a multi-level network model(cf. §3), and the execution of a complicated network restorationalgorithm (cf. §5). Of these, the second is the most difficult: thesimpler the performance measure defined on the system, the easier it isto deal with the size of the system (e.g. by some approximation orreduction technique) and its evolution in time, and the more detailedone can make the models for the individual components. So theformulation of §2 reflects a compromise, where we have basically tradedoff accurate representation of the network and its dynamics (2-statemodels for the components, assumption of steady state, assumption thatmultiple failures occur at a single instant in time and restorations aresingle events) in order to be able to handle the complexity of themechanisms involved in the performance measure. The remainder of thisdiscussion will clarify these assumptions.

3. Hierarchical Network Model

FIG. 16 shows two views of a network: a view of the “real” network, andthe view adopted by the performability model. Each view consists of anumber of levels, and the figure indicates the correspondence betweenthem. The real network view is divided into traffic, transport, andphysical layers. These are commonly-used terms in the networking field;we clarify later the sense in which we use them. The l.h.s. of FIG. 15uses the example of an IP network carried over an optical transportlayer, but this could also be an entirely different type of network,such as satellites communicating among themselves and with groundstations via microwave links, with the ground stations interconnected bya circuit-based network.

The performability model view has a demand level, a graph level(multigraph, in general), a component level, and a reliability level(not to be confused with network layers). The demand level specifies theamount (and possibly type) of traffic flowing from one node of the graphto another via the graph's edges. The graph is the level at which thenetwork routing and restoration algorithms operate, so graph edges haveassociated capacities and (routing) costs.

At the component level, each component corresponds to an independentfailure mechanism, as explained in §2; failure of a component may affecta whole set of graph-level elements. The components may, in fact, havean arbitrary number of failure modes; see §3.2 and §3.3. In general, theeffect of a component entering one of its failure modes on a graph nodeor edge is to change some of its attributes. For example, the capacityof an edge may decrease, or a node may become unable to perform routing.Depending on the final values of the attributes, e.g. edge capacity 0,the graph element may be considered “failed”. Finally, at thereliability level, failure modes of components are described simply bytheir mean times between failures and mean times to repair.

The model view in FIG. 15 b implicitly assumes that network restorationhappens at only one level. If we had a network employing optical-layerrestoration in addition to the IP-layer restoration, FIG. 15 b would beextended by adding another “optical” graph level between the componentand graph levels. Then components would be associated with elements atthe optical graph level and restorations would happen at both of thegraph levels.

3.1 Example Model for an IP-Over-Optical Network

Today's typical commercial packet network consists of IP routers whoselinks (between the routers) are transported by an underlying opticalnetwork. For such a network we illustrate how we model the layers FIG.15 a and then map them to a model in FIG. 15 b.

Traffic layer Based on an estimate of the peak or average trafficpattern, we create a matrix giving the demand (typically called “flows”)between each pair of routers. A demand has a rate, and possibly a typeassociated with it.

Transport layer nodes A network node represents an IP router whichcontains a data plane, a control plane, and ports. The data plane, orswitching fabric, is responsible for routing packets, while the controlplane computes routing tables, etc. We model the data and control planesas separate components, because newer routers have the ability tocontinue forwarding packets even when the control plane fails.

The nodes of the graph level model represent routers. At the componentlevel this node explodes into one data plane, one control plane, and onehardware and software upgrade component. When a data plane componentfails, all the graph edges incident to its node fail. When a controlplane component fails, the existing flows are unaffected, but the nodecannot participate in restoration. We also model components for eachport of the network node. Failure of a port component brings down thecorresponding edge(s). Finally, at the reliability level we specify anMTBF and an MTTR for each class of component.

Transport layer links Two network nodes may be connected by multipleparallel links. These are modeled at the graph level by parallel edges.For the purposes of IP routing, these parallel edges may form an“aggregate” link whose capacity is the sum of the capacities of theedges, and the nodes see only the aggregate links. When a constituentedge fails, the capacity of a link is reduced; the link fails if andonly if all its constituent edges fail. As stated earlier, edges fail ifthe corresponding port components fail. Edges can also fail because ofnetwork span failures, discussed next.

Physical layer spans We use the term “span” to refer to the networkequipment at the physical layer that carries the transport-layer links.We define a component for each network span. Failure of this componentaffects all graph-level edges which are routed over this span. When theoptical layer uses Dense Wavelength Division Multiplexing (DWDM), a spanconsists of a concatenation of point-to-point Optical Transport Systems(OTS) (more complex WDM systems have various optically transparent“add/drop” capabilities which, for simplicity, we do not discuss here).In turn, an OTS is composed of optical multiplexers and demultiplexers,optical amplifiers, and optical transponders. The length of a span andthe maximum permitted distance between optical amplifiers determine howmany OTSs need to be concatenated. To avoid proliferation of components,we do not define a component for each piece of equipment appearing in aspan, but represent the entire span by a single component; thiscomponent may affect multiple edges, as many as the number ofwavelengths.

A basic assumption about spans in typical transport networks is thatthey are considered to be working only if both directions of the spanare working. With this assumption, it is not difficult to compute thefailure probability of a span based on the failure probabilities of itsindividual components. However, at the reliability level we want a moredetailed characterization of a span, in terms of an equivalent failurerate Λ (MTBF) and an equivalent repair rate M (MTTR). Since bothdirections of a span must be working for the span to be operational, wecan represent the whole span by laying out the components of bothdirections into a series connection of independent blocks. The MTBF andMTTR of this connection are then calculated as explained in §3.3.

Other components We may also have other components, in addition to theabove types. For example, a bundle of fibers that is likely to failtogether because they are in a single conduit can be represented by acomponent that brings down all graph edges carried by this fiber bundle.Other types of catastrophic failures of entire sets of graph nodes andedges may be similarly represented.

Simplifying the graph-level access topology Commercial transportnetworks are usually partitioned into “access”, and a “backbone” or“core” segment. The topology of the access to the backbone or core isthe most complex part of the network, with many details that are verydifficult to model. Moreover, it undergoes frequent changes, as olderequipment is replaced by newer equipment and customer interfaces change.To analyze the availability and performance of the backbone segment ofour IP-over-Optical example network, at the graph level we create asimplified model (and hence the accompanying network “roll-up”approximation) of the access topology based on the point-to-pointdemands and assume only a single type of router port at the SONET OC-48(2.5 Gb/s) rate. The model is created as follows.

Given a backbone office, we go through all the demands originating fromit and try to fit them into as few ports as possible. These become the“customer-facing” ports of the location. We then distribute these portsinto groups of a certain size, which is a parameter. Each group isassigned to a virtual edge node, which has the same structure as anordinary node: it contains ports, both customer- and network-facing, adata plane, and a control plane. (The network-facing ports of an edgenode and the links to backbone nodes are determined by the networkdesign, as described in §6.)

3.2 Attributes for Graph Elements, Multi-Mode Components

A distinguishing feature of our model is that it allows graph nodes oredges to have any number of (possibly vector-valued) attributes, whosevalues can be set by components. Having such attributes at the graphlevel greatly increases the representational power of the model. As anexample, consider modeling what is known as a “composite network link”.Such a link represents a set of parallel edges which, for routingpurposes, is treated as a single edge, with capacity equal to the sum ofthe constituent capacities, and which fails if some of its constituentlinks fail and its composite capacity becomes less than a certainthreshold. Suppose we want to model a composite link that consists oftwo edges e₁, e₂, each of the same capacity c. In the real network e₁can fail because of a failed “port” component p₁, e₂ because of a failedport p₂, and both edges can fail because of a failed “span” component s,as shown in FIG. 16 a.

We can calculate F in a binary model by examining the states of p₁, p₂,s and implement the truth table of FIG. 16 b. But then the code for thetable is hard-wired into F, and if the network design changes, the codehas to be modified. On the contrary, our model allows us to define a2-element capacity attribute for the link, with the convention that thelink's capacity is the sum of these two attributes. Then if p₁ setscap[1] to 0 and p₂ sets cap[2] to 0, whereas s sets both to 0, thecomposite link can be correctly modeled. Graph element attributes alsohave other capabilities not covered in this example.

The use of components with multiple failure modes is not new inperformability analysis, so here we just mention an example in the IPnetworking context. Upgrading the software of a router results in ashort outage for the router. To minimize these outages, the commercialnetwork maintenance policy is that only a single router is upgraded at atime. We model these mutually exclusive “failures” by using a componentthat, for an n-router network, has n+1 modes, with mode i correspondingto upgrade of the ith router and the last mode representing no router inupgrade state.

Our model also allows the combination of multi-mode components and(vector) graph element attributes.

3.3 Markov Models for Components

Each component of the network model is represented by a simplecontinuous-time Markov process with states G and B (“good” and “bad”),and parameters λ and μ (failure and repair rates), as shown in FIG. 17.Note that the multi-mode model assumes that every failure is followed bya return to the working state. We use these models for components thatcorrespond to a single piece of equipment, as well as for componentsthat represent a connection of several pieces of equipment in series,e.g. the spans of §3.1. In the first case λ and μ are directly availablefrom the MTBF and MTTR of each mode, but in the second they arecalculated, as follows.

If we have a series connection of k independent binary blocks, thecorresponding Markov model has a single “up” state and 2^(k)−1 “down”states. We approximate this model by a 2-state model of the form shownin FIG. 17 a as follows. The 2-state model has states G and B. Gcorresponds to the single working state of the 2^(k)-state model, and Bto the set of failure states. Let π(G) be the steady-state probabilityof state G and π(B) the sum of the steady-state probabilities of thestates in B. Then the parameters of the 2-state model are

$\begin{matrix}{{{\Lambda = {\sum\limits_{i}\lambda_{i}}},{M = {\Lambda \frac{\pi (G)}{\pi (B)}}},{where}}{{\pi (G)} = {\prod\limits_{i}\; {\frac{\mu_{i}}{\lambda_{i} + \mu_{i}}.}}}} & (3.1)\end{matrix}$

and λ_(i) and μ_(i) are the failure and repair rates of components 1, .. . , k. When λ_(i)<<μ_(i), the above yields π(B)=q≈Σ_(i)q_(i), i.e., asexpected, the failure probability of the series system is close to thesum of the failure probabilities of the k blocks. This approximation hastwo desirable properties: (a) the corresponding steady-stateprobabilities of the original and aggregate models are equal, and (b)the exponential sojourn time of the 2-state model in each of its statesequals the approximately exponential sojourn time of the original modelin each of the sets G and B. (The derivation of (3.1) involves thedetails of the 2-state approximation.)

4. Performance Measures and Risk 4.1 Traffic Measures

Given a network specified by the hierarchical model of §3, we define thefollowing performance measures on every network state s:

t_(aff)(s)=traffic affected in state s,

t_(fail)(s)=traffic over failed edges in s, (4.1)

t_(lrt)(s)=traffic lost because of no route in state s,

t_(lcg)(s)=traffic lost because of congested edges in s.

Here “traffic” means the sum of the corresponding demands. A demand isaffected when an edge on its route fails. A demand fails when a link(multiedge) on its route fails. A failed demand is lost, and remains soif no path for it has been found after the restoration processcompletes.

If the network routing allows congestion, the total traffic lost becauseof congestion can be computed by finding the maximum possible total flowF that can be carried without congestion and subtracting it from thetotal traffic in the network. Let D be the set of all subdemands (ademand may be split routed over multiple paths; we refer to each ofthese as a “subdemand”), and D_(e) the set of subdemands using edge eunder the given routing. Also let f_(d) be the flow corresponding tosubdemand d. Then F is computed by solving the linear program

$\begin{matrix}{{\mathcal{F} = {\max {\sum\limits_{d \in \; D}{f_{d}\mspace{14mu} {subject}\mspace{14mu} {to}\mspace{14mu} {\forall{{e{\sum\limits_{d \in \; D_{e}}f_{d}}} \leq {c_{e}U_{c}}}}}}}},{{f_{d} \leq \upsilon_{d}};}} & (4.2)\end{matrix}$

where c_(e) is the capacity of edge e, U_(c)ε(0,1] is the congestionthreshold, and ν_(d) is the volume (size) of demand d. Finally,t_(lcg)=Σ_(d) _(ε) _(D) ^(ν) _(d) ^(−F) .

However, for IP networks, in general this provides a crude estimate ofpacket loss, because packet loss depends on a model for the traffic, aqueuing model for the nodes, and details of the protocol. E.g., TCP will“throttle down” traffic as soon as it detects congestion, so that packetloss is mitigated; in that case it is more accurate to call the t_(lcg)measure as computed above “loss in network bandwidth”.

4.2 Monotonicity

Whether a traffic measure is monotone or not (as defined in §2) dependsboth on the measure itself and on the routing/restoration method used bythe network. It is clear that the measure t_(aff) of (4.1) is monotoneincreasing, and so are t_(fail) and t_(lrt). This is so for both of theshortest-path and “optimal” restoration schemes considered in §5.However, in general the determination of monotonicity needs to be donecarefully because intuition can be misleading. For example, the naturalmeasure t_(res), the amount of traffic that is affected and successfullyrestored in state s, is not monotone: consider the example in FIG. 18with a unit demand on each of the heavy arcs. The traffic is defined tobe restored if there exists a non-failed path between the source anddestination (therefore the exact restoration method is irrelevant). Thefigure also shows that congested traffic is non-monotone, againirrespective of the restoration scheme. Hence the same holds for themeasure t_(lcg) of (4.1)

4.3 Incorporating Time

When computing the measures in (4.1) we also want to take into accountthe duration τ of a failure as well as the time τ_(r) needed forrestoration to complete. To do this, we will use the generic term “bad”traffic for the measure of interest. Let D_(bb) (s) be the set ofdemands in s that are bad after the failure but before restoration, andD_(ba) (s) the set of bad demands after restoration. These two sets mayintersect. If t_(bb) (s) and t_(ba) (s) are the traffic ratescorresponding to D_(bb) (s) and D_(ba) (s), the bad traffic as afunction of time is depicted in FIG. 19. In practice, τ_(r) is usuallymuch less than τ. The duration τ of state s is distributed exponentiallywith mean {tilde over (w)}_(s), the mean waiting time of the Markovprocess in s (recall §3.3). It can be shown that the expectation of thebad traffic in this state with respect to τ is

t _(b)(s)=t _(bb)(s)+(t _(ba)(s)−t _(bb)(s))e ^(−τ) ^(r)^(/{tilde over (w)}) ^(s) .   (4.3)

For example, when “lost because of congestion” is substituted for “bad”and the appropriate “before” and “after” sets of demands are computed,the result of (4.3) is to be taken as the value of t_(lcg) (s) in (4.1).

The analysis just given clearly involves some idealizations. Forexample, the set of demands D_(bb) (s) actually changes during therestoration process; hence t_(bb) also changes and is not constant asimplied by FIG. 19.

4.4 Risk and the Pareto Frontier

Given a set of measures such as the ones in (4.1), one way to assess theperformability of a network is to check if their CDFs satisfy certainrequirements, the performability guarantees of (2.2). An alternative tolooking at performability guarantees is to examine the space of exploredfailures from the viewpoint of risk.

Given a performance measure, let the risk of a failure with respect tothis measure be the product of the probability of the failure and thevalue of the measure. With an m-dimensional measure, the risk associatedwith a failure has m aspects, or attributes. An immediate question is“what are the riskiest failures for the network?” Because of themultiplicity of attributes, the set of network states cannot be totallyordered with respect to risk, and there will usually not exist a failurethat is worse than all the others. But we can say that a failure state sdominates (is worse than) another state s′ if the risk attributes of sare ≧the corresponding attributes of s′, with strict inequality for atleast one attribute. Given a set of network states, we can eliminatefrom consideration all states whose risk is dominated by that of someother state. What remains is known as the Pareto boundary or Paretofrontier of the risk set. The frontier can be extracted from the set bywhat is known as a “Pareto filter”.

The filtering problem is: if V is a set of d-dimensional vectors inR^(d), we want to eliminate from V all points v that are dominated byanother point uεV:u dominates v, denoted u

v, if and only if u_(i)≧v_(i) for all i with strict inequality for atleast one i. Suppose that determining whether x

y takes d units of time. Then given n points, a straightforwardalgorithm for eliminating all dominated points takes time O(dn²). A muchbetter algorithm for d=2, requiring only O(n log n) time, is given asfollows. Denote a point by (x_(i), y_(i)) and sort the points in orderof increasing x, with increasing y for equal values of x_(i). Withoutloss of generality, let the result be (x₁, y₁), (x₂, y₂), . . . ,(x_(n), y_(n)). Then it is clear that any point can be dominated only bya point to its right. Starting from the end, (x_(n−1), y_(n−1)) isdominated by a point on its right if and only if y_(n−1)<y_(n).(x_(n−2), y_(n−2)) is dominated by a point on its right if and only ify_(n−2)<max(y_(n−1), y_(n)), etc. Finally, (x₁, y₁) is dominated by somepoint to its right if and only if y₁<max(y₂, . . . , y_(n)). Denotingthe maxima by m_(i), the algorithm finds them in linear time by making abackward pass through the list to determine m_(n−1)=y_(n),m_(n−2)=max(y_(n), y_(n−1)), m_(n−3)=max(y_(n), y_(n−1), y_(n−2)), . . ., m₁=max(y_(n), . . . , y₂). (The filtering problem may also be solvedin O(n log n) time with a more general bi-criterion shortest pathalgorithm, but the procedure we described here is much simpler.)

5. Network Restoration in Performance Measures

We have implemented many of the concepts described in previous sectionsin software to evaluate commercial networks. This section describes therouting methods we implemented to model the IP layer for commercialIP-over-optical layered networks. However, we have also applied thesemethods to other layered networks. In each network state we have a newgraph (a set of nodes fail, a set of edges fail or their capacities arereduced, etc.), and a set of demands affected by a failure that needs tobe rerouted. The performance measures of §4 depend partly on how weselect new routes for the failed demands. We use two schemes forselecting paths during network restoration:

1. Shortest-path routing, which models OSPF (Open Shortest Path First)routing where packets are sent over the shortest (minimum cost) paths.The path computation does not depend on the capacities or utilizationsof the links. Note that the other main IP-layer routing protocol, IS-IS,would behave similarly in our context.

2. Optimal routing, which selects a set of minimum-cost paths thatmaximizes the amount of restored traffic while respecting the linkcapacities. This models a scheme where a centralized server has completeknowledge of the entire network and computes optimal routing paths.OSPF-TE (MPLS) may be considered to be an approximation to this.

The shortest-path routing, because it does not take link capacities intoaccount, is likely to result in more congested links and thus largervalues of t_(lcg). On the other hand, it will have a lower value oft_(lrt) because it checks only for connectivity, as opposed to a pathwith sufficient capacity.

Clearly, the above are idealized representations of real network routingand restoration protocols, in which many details are not taken intoaccount. Notably, we do not model in detail the full set of complicatedmessage exchanges (known as “signaling”) that take place among networknodes to implement the routing algorithm.

5.1 Min-Cost Routing

We first construct a graph G=(V, E) where V is the set of network nodesand E represents the OSPF links. The link costs are equal to the OSPFadministrative weights. The capacity of a link (multiedge) is equal tothe sum of the edge capacities between this pair of nodes (recall §3.1).Capacities are not used for the minimum-cost (min-cost or shortest-path)computation, but are used for determining the set of congested links forcalculation of t_(lcg).

If there is more than one min-cost path between a pair of nodes, webalance the traffic as follows: for each source and destination pair (s,d) we compute NextHops(s, d), the set of neighbors of s that are on atleast one min-cost path from s to d. Then we distribute all packets froms to d equally among all nodes in NextHops(s, d), and repeatrecursively. To compute the all-pairs min-cost matrix we use theFloyd-Warshall algorithm). For a graph with n nodes and m edges, thecomplexity of this algorithm is O(n³).

5.2 Near-Optimal Routing

The network is specified as a directed graph with edges that havecapacities and costs and a set of indivisible demands. The idealizedproblem that we want to solve is optimal restoration, i.e. “route asmuch total demand as possible subject to the capacity constraints, andso that the total cost is minimized”.

This problem involves optimizing both the quantity of traffic routed andthe cost (good-ness) of the routing. In network flow terminology, wehave an integer min-cost max-multicommodity flow problem, i.e. among themaximal flows find one with least cost (the cost of a flow, equal toΣ_(i) {flow over edge i×cost of edge i}).

Even very simple versions of this problem are hard. For example,deciding whether just two demands can be routed in a graph with edgesall of the same capacity and no costs is NP-complete. But if theintegrality (indivisibility) requirement on the edge capacities andflows is removed, min-cost multicommodity flow is solvable in polynomialtime via linear programming.

We find an approximate solution to the optimal restoration problem byusing a greedy heuristic which reduces the problem to a set of1-commodity flow problems, and at the same time takes care of theintegrality requirement. This is depicted in FIG. 20, where we employ Rrandom orderings of N demands. Note that if an ordering is such that allof the single-commodity flow problems are feasible, it does notnecessarily yield a solution of least cost. Given that optimalrestoration can be solved only approximately, the greedy heuristicsuggests the possibility of separating the quantity and cost aspects ofthe performance measure, and stopping the algorithm if a solution thatroutes all of the offered demand is found, no matter what its cost. Ourimplementation includes this option; it may be argued that such asolution corresponds better to what a realistic, non-centralized networkrouting protocol could achieve.

Finally, the heuristic lends itself well to speedup by parallelization.Given a set of M processors, R/M permutations are given to each, and,when finished, each processor reports its best result to thecoordinating processor. Our implementation includes the option ofparallel evaluation, using the Parallel Virtual Machine [PVM]infrastructure.

5.3 Speeding Up the Routing Computation to Enable Large State SpaceAnalysis

It is not unusual for a network performability analysis to generatehundreds of thousands of states, and paths for the network demands needto be computed in every one of these states. We describe someimprovements in the overall efficiency of the path computation thatexploit the fact that when states are generated in order of probability,the state sequence exhibits a certain locality: typically, successivenetwork states differ in only one component. In the case of min-costrouting, this implies that the all-pairs min-cost (classicalshortest-distance) matrix typically changes little between adjacentstates. One way to take advantage of this is via a dynamic graphalgorithm. There are several such algorithms with strong worst-caseasymptotic guarantees, but they are not easy to implement. In contrast,the improvements we describe below are very easy to implement and resultin near-optimal running time for our specific application.

Improvement 1: affected node pairs Let D₀ be the all-pairs min-costmatrix for the network in the perfect (no failure) state. We want toreuse D₀ as much as possible in any state by computing the set of nodepairs for which the min-cost has changed. Because there is no easy wayof computing this set we use a filter to compute a superset. For anynode pair, if the minimum-cost path between these nodes contains atleast one failed edge, we put the node pair in the setAffectedNodePairs. If a node pair is not in this set, its min-cost pathdoes not change (all of the edges in the shortest paths are there, andno new edges have been created). Then we run a modified Floyd-Warshallalgorithm where the two innermost loops range over the nodes inAffectedNodePairs. FIG. 21 gives the pseudo-code. Note that theall-pairs min-cost path matrix is defined to be M (i, j)=the cost of themin-cost path from node i to node j. To establish the correctness ofthis algorithm, it remains to show that the min-cost for node pairs inAffectedNodePairs, which has either stayed the same or increased, iscorrectly computed. The only difficult part of this proof is to arguethat the initialization in line 1 of D to D₀ (instead of initializationto edge costs) does not affect the correctness of the Floyd-Warshallalgorithm. We establish this by a general proposition:

Proposition. For any graph G(V, E), construct a graph G′ (V, E′) asfollows. Initialize E′ to E. Then pick an arbitrary pair of nodes u, vand add an edge (u, v) to E′ with cost equal to the min-cost pathbetween u and v in G. Repeat this step for an arbitrary number of nodepairs. Then the min-cost matrices for G and G′ are identical.

This can be proved by induction on the number of extra edges added toE′. For each extra edge, a path of the same cost existed in the graphand thus none of the costs change because of the addition of this edge.Because the effect of initialization is the same as adding an extra edgeof cost equal to the minimum cost path, this also completes our proof ofthe dynamic all-pairs min-cost algorithm.

Given the matrix D₀, the set of AffectedNodePairs can be computed inO(n²·(number of failed edges)) time by cycling over each pair of failededge (u, v) and node pair (s, d) and checking if D₀ [s, u]+EdgeCost[u,v]+D₀ [v, d]=D₀ [s, d]. It takes an additionalO(n²·|AffectedNodePair(s|) steps to run the modified Floyd-Warshallalgorithm. Thus the overall complexity is O n²·(no. of failededges+|AffectedNodePairs|)

Improvement 2: matrix caching Both the number of failed edges and thesize of AffectedNodePairs depend on the size of the failure. Our nexttrick is to “reduce” the case of multiple component failures to the caseof single component failures. This reduction is possible quite oftenbecause of the locality in the sequence of states produced by thealgorithms of §2.2. When the reduction is possible, its effect is that“number of failed edges+size of AffectedNodePairs” corresponds to asingle component failure, significantly improving the speed of thealgorithm in FIG. 21.

Based on these observations, we modify the algorithm to maintain a cacheof (set, matrix) pairs. In a state corresponding to a set of failedcomponents C, we first check if the cache contains an entry (C₁, D₁)such that C₁ is missing exactly one element of C. If such an entry isfound, the algorithm of FIG. 21 is run with D₀=D₁ and as if a singlecomponent c=C\C₁ has failed. If there is no match for C in the cache, weconstruct C₁ by copying C and removing the component which is leastlikely to fail from it. Then we compute the distance matrix D′₁corresponding to C₁ and add (C₁, D′₁) to the cache.

In our simulations of this scheme with a cache size of 25 andleast-recently used re-placement policy, we found a cache hit rate ofnearly 99%. Moreover, as we searched the cache from top to bottom, about80% of the time we had a match among the first 2 entries, and about 90%of the time a match among the first 5 entries. Thus the averagecomplexity (per state) of the all-pairs min-cost paths algorithm becomesa small constant times n².

6. Applications and Experimental Results

Here we report the results of running our tool on three network designs,variations on the IP-over-optical backbone network of a large InternetService Provider.

Each design consists of a number of backbone locations. Each locationhouses one or two backbone routers (BRs), and has a set of associatededge routers (ERs). The ERs are connected to the BRs within thelocation. Each BR is connected to the other BR in the location, and to aset of BRs in other locations; see FIG. 6.1. The ERs are divided intotwo groups: local and remote. We assume that the physical layer routings(spans) of the links connecting BRs and local ERs and BRs do not fail,although these pairs of nodes can still get disconnected if thecorresponding router ports fail. The remaining links fail if at leastone of their underlying spans fails (recall the model of §3.1). For ourexperiments we use component failure probabilities derived from acombination of field and vendor data. The location-to-location trafficmatrix is given (produced from a “gravity model” algorithm based on IPlink loads). We study three variations of a network design, whose basicstructure is depicted in FIG. 22:

The “fat” design. Each location has two BRs, and each ER is connected toboth the BRs in its associated location. The network has enough capacityto protect against any single span, port, or BR failure.

The “thin” design. This is similar to the fat design except that we donot allocate restoration capacity for BR failures.

The “skinny” design. Here we have only one BR per location, and each ERis connected to the single BR in the location. The network has enoughcapacity to protect against any single span and port (but no BR)failure.

All of these networks use OSPF-style restoration, and the designs usethe “80/95” rule, namely that link packet utilization should be below80% under no-failure conditions, and not exceed 95% after a failure.Specifically, the edge capacities are assigned so that each networkmeets the 80/95 rule for a certain set of single failures. This approachis common among most large ISPs, although the utilization levels andnetwork design specifics differ. After a failure that is not in the“covered” set, some links may be utilized over 95% and it is evenpossible that some flows may not be restored. Our goal is toquantitatively evaluate effect of using reduced network capacity andfewer routers on the performability of the network. We also evaluate theeffects of some improvements to routers, namely “hitless”software/hardware upgrades and N:1 port protection. For each networkdesign we compute the CCDF (complementary CDF) of two measures: trafficlost because of no route, and traffic lost because of congestion, thet_(lrt) and t_(lcg) of (4.1). We define the lost traffic t_(lost) to bethe sum of these two. To protect proprietary information, we havemultiplied the actual probabilities (y-axes) in all our plots by a fixedconstant.

6.1 Lost Traffic

FIG. 23 a shows the bounds on the lost traffic t_(lost) for the fatdesign obtained by (2.4) after generating about 400,000 network states.The lower bound assumes that none of the traffic is lost in theunexplored state space, and the upper bound assumes that all of thetraffic is lost in the unexplored state space. In our runs, theunexplored failure space consists of some double, triple, and quadruplefailure states and all of the failures of larger sizes. If we weightthese states according to their probabilities, most of the unexploredprobability mass is concentrated on double and triple failure states.Based on empirical knowledge of network restoration performance in thedouble and triple failure states, the precise lost traffic metric iscloser to the lower bound. Obviously, this conclusion changes if weexplore more states, or investigate a different type of network. Forsimplicity, in all subsequent plots we show only the lower bound curve.FIG. 23 b shows lost traffic for all three network designs. For all ourreported results in this paper we will show only the t_(lost) curves.Note that the t_(lrt) component (lost traffic because of no path)depends only on the network topology, and therefore the fat and the thindesigns have identical performance with respect to t_(lrt). However, asFIG. 23 b shows, the fat and the thin designs have very differentcharacteristics when compared on the basis of total lost traffic, whichincludes t_(lcg) the traffic lost because of congestion.

In all these CCDF graphs, the plots are smoothed versions ofstaircase-like functions. The abrupt changes in slope of the CCDF curvesare due to the discrete nature of the probability space and the complexmapping from this space to the value of the measure. In particular, therelatively flat segments are not caused by any particular set offailures, but by the absence of failures that, via the discrete mappingsfrom the physical to the traffic layer, cause traffic loss of a certainmagnitude.

One may note in FIG. 23 b that the skinny network gives a betterguarantee than the thin network at low values of t_(lost). The reason isthat the skinny network has fewer components (e.g. router ports), andthus a higher probability of being in the no-failure state. As thefraction of lost traffic increases, the better restoration behavior ofthe thin network translates into better performance. On the other hand,the fat network is superior everywhere because its restoration behavioris so much better than that of the thin and skinny networks.

6.2 Router Enhancements

Here we look at two possible improvements to the reliability of routersand assess the effect of these improvements on the entire network fromthe viewpoint of lost traffic.

6.2.1 Hitless Upgrades

FIG. 23 assumes that whenever a router's software or hardware isupgraded, the router is taken out of service. We assume three 30-minutesoftware upgrades and one 60-minute hardware upgrade per year for eachrouter, and model the state space using multi-mode components such thatat most one BR from a given backbone location gets upgraded at a time(recall §3.2). A new generation of commercial routers provides “hitless”software upgrades, and we want to assess the resulting performabilityimprovement. FIG. 24 shows that the improvement is dramatic. In fact,the performance of the “thinned-down” designs becomes better than theperformance of the fat design with normal upgrades! One caveat is thatthese results were obtained assuming a constant traffic matrixthroughout a day, whereas in reality router upgrades are often scheduledduring low-traffic periods. If such an upgrade policy were in effect,the difference in the performance of the three designs would be smallerthan what FIG. 24 indicates.

6.2.2 Port Protection

Here we investigate the effect of adding N:1 protection on router ports.We define groups of N ports ahead of time, each with an (N+1)thprotection port. When a port fails and the protection port in its groupis working, then the link is switched to the protection port, whichtakes over the role of the failed port. If multiple ports within thesame protection group fail, at most one of the failed ports can berestored; the rest remain failed.

Without port protection, the probability that a port does not fail is1−q. With protection, a port stays up in any state in which it has notfailed; the set of all these states has probability 1−q. The port alsostays up in any state in which it has failed but every other port,including the protection port in its group, is up; this set of stateshas total probability q(1−q)^(N). Hence

Pr(a protected port stays up)=(1−q)+q(1−q)^(N)≧1−Nq ².

This shows that for typical (small) values of q, the effect ofprotection is, surprisingly, almost independent of the protection groupsize N. FIG. 25 shows the results for the fat design. It can be seenthat without hitless router upgrades, port protection is not veryeffective. This is because a port can go down either because the portitself fails, or because the whole router is down. Port protection helpsin the first case, but not in the second. In our model, the router beingdown is an order of magnitude more likely than one of its router portfailures, so it makes little difference whether or not ports areprotected. However, router upgrades are the major contributor to arouter being down. If we make the upgrades hitless, the probability of arouter being down no longer dominates the probability of a port failureand FIG. 25 b shows that port protection does make a significantdifference. In both cases, the benefits of port protection areconcentrated near low percentages of lost traffic. The higherpercentages of lost traffic result from failure states (typicallyinvolving complete router failures and fiber cuts) where port protectionmakes little difference.

The conclusion for the network in this case study is that softwareupgrades are a major contributor to network downtime. If the upgradeswere made hitless, port protection would further improve the network'sperformability, and there is even a promise of being able to reducerestoration capacity and still maintain the same performability levels.

6.3 Risk Assessment

In §6.1 and §6.2 we evaluated three network designs by looking atperformability guarantees, recall (2.2), for the lost traffic measure.Another, complementary, assessment consists in examining the space ofexplored failures from the viewpoint of risk and the Pareto frontier.

As explained in §4.4, risk is defined with respect to a particulartraffic measure: at each network state, the value of the measure ismultiplied by the probability of the state. With the 4-dimensionaltraffic measure of (4.1), the risk associated with a failure has 4attributes. FIG. 26 a shows a scatter plot of two of these attributesfor the fat network design, t_(lcg) and t_(lrt) risk. There are about400,000 points in the plot, and each point (x, y) denotes a failure withno-route risk x and congestion risk y. Evidently the risks vary by manyorders of magnitude, but the upper right “corner” of the scatter plot,in this case above the point (10000,10000), contains the worst failureswith respect to both risk attributes. These are the failures that onewould want to address first when contemplating improvements to thenetwork; they lie on Pareto frontier of the failure set. FIG. 26 b showsthe result of running the Pareto filter on the failures of FIG. 26 athree times in succession, each time eliminating the extracted frontierfrom the set. The first and second frontiers consist of a single pointeach, corresponding to the failure of a major virtual edge router(recall §3.1) of the ISP, while the third frontier consists of thefailures of another two major virtual edge routers.

Conclusion

The foregoing is merely illustrative. For example, although theinvention has been illustrated in the context of packet networks, theprinciples of the invention are equally applicable to other kinds ofnetworks. For example, the invention could be used to determine aperformability characteristic for circuit-switched networks. In such anapplication of the invention, a typical performance measure would be “Isthere a path available between source A and destination B?” Anothermight be, “Is there a path available between source A and every possibledestination B, C, D, etc” Another might be “Is there a path availablebetween each possible source/destination pair.” In each of these cases,the performance measure would have the value of “1” or “0” for eachdifferent network state depending on whether or not the path(s) inquestion are available in that network state. The performabilitycharacteristic, then, would be in an indication of the extent to whichit is likely, across all network states considered, that the path(s) inquestion would, in fact, be available.

It will thus be appreciated that those skilled in the art will be ableto design other arrangements and methods which, although not explicitlyshown herein, embody our inventive principles and are thus within theirspirit and scope.

1. A method for computing a performability characteristic for acommunications network that comprises a plurality of network componentsincluding a plurality of interconnected links, the method comprisingcomputing a performance measure associated with each of a plurality offailure states of the network, each of at least ones of the failurestates including at least one failed network component, and computingthe performability characteristic based on the performance measurescomputed for the plurality of failed network states, the performancemeasure being computed based on at least one attribute of one or more ofthe network components that is other than the traffic-handling capacityof one of the links.
 2. The method of claim 1 wherein the at least oneattribute is an administrative weight of the one or more of the links.3. The method of claim 1 wherein the at least one attribute is packetloss on one or more of the links.
 4. The method of claim 1 wherein theat least one attribute is latency of one or more of the links.
 5. Themethod of claim 1 wherein the performability characteristic is afunction of the product of a) each of the performance measures with b) aprobability that the network is in the associated failed network state.6. The method of claim 1 wherein the performability characteristic isone of a) lost traffic, b) packet delay, and c) path availability.
 7. Amethod for computing a performability characteristic for acommunications network that comprises interconnected links, the methodcomprising computing a performance measure associated with each of aplurality of failure states of the network, each of at least ones of thefailure states including at least one failed network component, theperformance measure being computed based on attributes of edges of alogic level graph of the network, each of the edges of the logic levelgraph representing a route through the network over one or morecorresponding ones of the links, the attributes of the edges beingdetermined at least by attributes of the one or more corresponding linksin said each of the failed network states, and computing theperformability characteristic based on the performance measures computedfor the plurality of failed network states, the at least one attributebeing other than the traffic-handling capacity of said one or morecorresponding links.
 8. The method of claim 7 wherein the performabilitycharacteristic is a function of the product of a) each of theperformance measures with b) a probability that the network is in theassociated failed network state.
 9. The method of claim 8 wherein the atleast one attribute is administrative weight of the one or morecorresponding links.
 10. The method of claim 8 wherein the at least oneattribute is packet loss of the one or more corresponding links.
 11. Themethod of claim 8 wherein the at least one attribute is latency of theone or more corresponding links.
 12. The method of claim 8 wherein theat least one attribute is route length over the one or morecorresponding links.
 13. The method of claim 8 wherein wherein theperformance measure is computed further based on attributes of nodes ofthe logic level graph, each of the nodes of the logic level graphrepresenting a associated network element connected to at least one ofthe links, and wherein the at least one attribute is an attribute of thenetwork element represented by one of the nodes of the logic levelgraph.
 14. The method of claim 13 wherein the network element is aswitching element.
 15. The method of claim 14 wherein the switchingelement is a packet router.
 16. The method of claim 14 wherein the atleast one network element attribute is the extent to which the networkelement is able to support new circuits through the network.
 17. Themethod of claim 8 wherein the performability characteristic is one of a)lost traffic, b) packet delay, and c) path availability.
 18. Acomputer-readable medium on which are stored instructions that areexecutable by a processor to carry out the method defined by claim 1.